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Providing a set of database images 



Providing positive and negative example images 



For each databt 


ise image, computing a relevance score based on a 


simSaiity of the da 


:abase image to the positive eiample image considering 




relevant features 
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Creating a list of relevant images 
comprising the Nbl images having 
the highest similarity score among the set of database images 



I 



Providing discriminating features 
allowing to differentiate between 
the positive and negative example images 



3: 



For each relevant image in the list of relevant images, computing a 
iiscrimination score based on the similarity of the relevant image with the 
positive example image considering the discriminating features and on a 
dissimilarity of the relevant image with to the negative example image 
considering the discriminating features 



Selecting the Nb2 images having 
the highest discrimination score among 
the list of relevant images 



(57) Abstract: Although negative example can be highly useful 
to better understand the user's needs in content-based image re- 
trieval, it was considered by few authors. A content-based image 
retrieval method according to the present invention addresses 
some issues related to the combination of positive and negative 
examples to perform a more efficient image retrieval. A rele- 
vance feedback approach that uses positive example to perform 
generalization and negative example to perform specialization 
is described herein. In this approach, a query containing both 
positive and negative example is processed in two general steps. 
The first general step considers positive example only in order to 
reduce the set of images participating in retrieval to a more ho- 
mogeneous subset Then, the second general step considers both 
positive and negative examples and acts on the images retained 
in the Orst step. Mathematically, relevance feedback is formu- 
lated as an optimization of intra and inter variances of positive 
and negative examples. 



Best Available Copy 



wfto04/015589 Al lllllillilllilllttlllllllilllllllillllllliiM^ 



Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 
European patent (AT, BE, BG, CH, CY, CZ, DE, DK, EE, 
ES, n, FR, GB, GR, HU, IE, FT, LU, MC, NL, PT, RO, 
SE, SI, SK, TR), OAPI patent (BF, BJ, CP, CG. CI. CM, 
GA, GN, GQ, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— with international search report 



— before the expiration of the time limit for amending the 
claims and to be republished in the event of receipt of 
amendments 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



wo 2004/015589 ^^PCT/CA2003/001215 



TITLE OF THE INVENTION 

CONTENT-BASED IMAGE RETRIEVAL METHOD 
FIELD OF THE INVENTION 

[0001] The present invention relates to digital data retrieval. More 

specifically, the present Invention is concerned with content-based image 
retrieval- 

BACKGROUND OF THE INVENTION 

[0002] With advances in the computer technologies and the advent 

of the World-Wide Web, there has been an explosion in the quantity and 
complexity of digital data being generated, stored, transmitted, analyzed, and 
accessed. These data take different fonms such as text, sound, images and 
videos. 

[0003] For example, the increasing number of digital images 

available brings the need to develop systems for efficient image retrieval which 
can help users locate the needed images in a reasonable time. Some of these 
retrieval systems use attributes of the images, such as the presence of a 
particular combination of colors or the depiction of a particular type of event. 
Such attributes may either be derived from the content of the image or from its 
surrounding text and data. This leads to various approaches in image retrieval 
such as content-based techniques and text-based techniques. 

[0004] in any case, when an Image retrieval system returns the 

results of a given query, two problems often arise: noise and miss. Noise arises 
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when images which don't correspond to what the user wants are retrieved by 
the system. i\/liss is the set of images corresponding to what the user wants 
which have not been retrieved. These two problems originate from 
imperfections at different levels. Indeed, it may not be easy for the user to 
formulate an adequate query using the available images, either because none 
of them correspond to what the user wants or because the user lacks sufficient 
knowledge of imagery details to articulafe Image features. Also, it has been 
found difficult to translate the user's needs and specificities in terms of Image 
features and similarity measures. 

[0005] More specifically in the case of content-based image 

retrieval, one can distinguish many ways of formulating queries. Early systems 
such as QBIC, which is described by Flicker et al. in "Query by image and 
video content. The QBIC system" in IEEE Computer Magazine, 28:23-32, 
1995, prompt the user to select image features such as color, shape, or texture. 
Other systems like BLOBWORLD which is described by Carson et al. In "A 
system for region-based Image Indexing and retrieval" from the International 
Conference on Visual Information Systems; pages 509-516, Amsterdam, 
1999, require the user to provide a weighted combination of features. 

[0006] However, a drawback of such content-based image retrieval 

techniques is that it is generally difficult to directly specify the features needed 
for a particular query, for several reasons. A first of such reasons is that not all 
users understand the Image vocabulary (e.g. contrast, texture, color) needed to 
formulate a given query. A second reason is that, even If the user is an Image 
specialist. It is not easy to translate the images the user has In mind Into a 
combinatiori of features. 
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[0007] An alternative approach is to allow the user to specify the 

features and their corresponding weights implicitly via a visual Interface known 
in the art as "query by example". Via this process, the user can choose images 
that will participate in the query and weight them according to their 
resemblance to the Images sought. The results of the query can then be refined 
repeatedly by specifying more relevant images. This process, referred to in the 
art as "relevance feedback" (RF), is defined Rui et al. in "Content-based image 
retrieval with relevance feedback in MARS" from the IEEE International 
Conference on Image Processing, pages 815-818, Santa Barbara, Califomia, 
1997, as the process of automatically adjusting an existing query using 
Information fed back by the user about the relevance of previously retrieved 
documents. 

[0008] Relevance feedback is used to model the user subjectivity in 

several stages. First, It can be applied to identify the Ideal images that are in 
the user's mind. At each step of the retrieval, the user Is asked to select a set of 
images which will participate in the query; and to assign a degree of relevance 
to each of them. This Information can be used In many ways in order to define 
an analytical form representing the query intended by the user. The ideal query 
can then be defined Independently from previous queries, as disclosed in 
"l\/lindreaden Query databases through multiple examples" in 24th International 
Conference on Very Large Data Bases, pages 433-438, New York, 1998 by 
Ishikawa ef al. It can also depend on the previous queries, as in the "query 
point movement method" where the ideal query point is moved towards positive 
example and away from negative example. This last method is explained by 
Zhang ef al. in "Relevance Feedback in Content-Based Image Search" from 
the 12th International Conference on New Infomnation Technology (NIT) in 
Beijing, Uay 2001 . 
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[0009] Relevance feedback allows also to better capture the user's 

needs by assigning a degree of importance (e.g. weight) to each feature or by 
transfomning the original feature space into a new one that best conresponds to 
the user's needs and specificities. This is achieved by enhancing the 
importance of those features that help in retrieving relevant images and 
reducing the importance of those which do not. Once the importance of each 
feature is detennined, the results are applied to define similarity measures 
which con-espond better to the similarity intended by the user in specific current 
query. 

[0010] The operation of attributing weights to features can also be 

applied to perform feature selection, which is defined by Kim et al. in "Feature 
Selection in Unsupervised Learning via Evolutionary Search" from the 6th ACM 
SIGKDD International Conference on Knowledge Discovery and Data Mining 
(KDD-00), pages 365—369. San Diego, 2000, as the process of choosing a 
subset of features by eliminating redundant features or those providing little or 
no predictive information. In fact, after the Importance of each feature is 
determined, feature selection can be performed by retaining only those features 
which are important enough; the rest being eliminated. By eliminating some 
features, retrieval perfomnance can be Improved because. In a low-dimension 
feature space, it is easier to define good similarity measures, to perform 
retrieval in a reasonable time, and to apply effective indexing techniques (for 
more detail, see "Web Image Search Engines: A Survey. Technical Report H° 
276, Universite de Sherbrool<e, Canada, December 2001 , by Kherfi et al.). 

[0011] Relevance feedback using positive examples is very well 

known In the art. For example, Ishikawa ef al. define a quadratic distance 
function for comparing images. Considering a query consisting of N images, 
each Image represented by an l-dimenslon feature vector x„ =[x„i,...,x^f , 
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Where T denotes matrix transposition and considering also that the user 
associates each image participating in the query with a degree of relevance n„ 
which represents its degree of resemblance with the sought images Ishikawa 
Bt al. compute two parameters, namely the ideal query q = f and the 

ellipsoid distance matrix W. that minimize the quantity D given in Equation (1). 
which represents the global distance between the query images and the ideal 
query: 



N 
n=i 

A drawback of the method proposed, by Ishikawa et al. is that it doesn't support 
the negative example. 

[0012] Rui et a/.(2) in "Optimizing Leaming In Image Retrieval". 

IEEE International Conference On Computer Vision and Pattern Recognition, 
Hilton Head, Sc. USA. 2000 disclose a method where each image is 
decomposed into a set of / features, each of which represented by a vector of 
reals. represents the i*^ feature vector of the n* query Image and ;r„the 
degree of relevance assigned by the user to the n* Image. It is assumed also 
that the query consists of N images. For each feature i. the ideal query vector 
q, , a matrix W| and scalar weight ui which minimize the global dispersion of the 
query images given by Equation (2) are computed. Minimizing the dispersion of 
the query images alms at enhancing the concentrated features, i.e.. features for 
which example Images are close to each other. 

J A' 

»=1 n=l /o\ 
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[0013] In "Efficient Indexing. Browsing and Retrieval of Image/Video 

Contenr, PhD thesis, Department of Computer Science, University of Illinois at 
Urbana-Champaign, 1999, Rui eta/ (3) propose to use a similar model but with 
negative degrees of relevance assigned to negative example images. A 
drawbacic of this model, is that it leads to neglect the relevant features of 
negative example, so that negative example will be confused with positive 
example. 

[0014] It is to be noted that, while many studies have focused on 

how to learn from user interaction in relevance feedback, few of them evoked 
the relevance of negative example. However, negative example can be useful 
for query refinement since it allows to detennine the images the user doesnt 
want in order to discard them. Indeed, MUller et al. shows, in "Strategies for 
Positive and Negative Relevance Feedback in Image Retrieval.". Technical 
Report N° 00.01, Computer Vision Group, Computing Center, University of 
Geneva, 2000, that, using only positive feedback, yields major improvement 
only at the first feedback step, while improvement is remarkable for the four first 
steps with positive and negative feedback where the results continuously get 
better. 

[0015] Relevance feedback with negative example may also be 

useful to reduce noise (undesired images that have been retrieved) and to 
decrease the miss (desired images that have not been retrieved). Indeed, after 
the results of a given query are obtained, the user can maintain the positive 
example images and enrich the query by including some undesired images as 
negative example. This Implies that images similar to those of negative 
example will be discarded, thus reducing noise. At the same time, the 
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discarded images will be replaced by others which would have to resemble 
better what the user wants. Hence, the miss will also be decreased. 
Furthermore, the user can find, among the recently retrieved images, more 
images that resemble what the user needs and use them to formulate a new 
query. Thus, the use of negative example would help to resolve what is called 
the page zero problem, i.e., that of finding a good query image to initiate 
retrieval. By mitigating the page zero problem, it has been found that the 
retrieval time is reduced and the accuracy of the results is improved (see 
Kherfi et a/.). It is also to be noted that relevance feedback with negative 
example is useful when, in response to a user feed-back query, the system 
returns exactly the same images as in a previous iteration, Assun;iing that the 
user has already given the system all the possible positive feedback, the only 
way to escape from this situation is to choose some images as negative 
feedback. 

[001 6] Consider the interpretation of results for content-based image 

retrieval methods involving negative example, one can distinguish two 
categories of models. In the first category, the positive example images are 
selected by the user; however, the negative example images are chosen 
automatically by the retrieval system among those not selected by. the user. In 
the second category, both positive and negative example images are chosen 
by the user. 

[0017] Muller ef a/, describe a content-based image retrieval 

method from the first category. Concerning the initial query, they propose to 
enrich it by automatically supplying non-selected images as negative example. 
For refinement, the top 20 images resulting from the previous query as positive 
feedback are selected. As negative feedback, four of the non-retumed images 
are chosen. The MQIIer method allows refinement through several feedback 
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Steps; each step aims at moving the ideal query towards the positive example 
and away from the negative example. More specifically, this Is achieved by 
using the following formula proposed by Rocchio in "Relevance Feedback in 
Information Retrieval" in SMART Retrieval System. Experiments in Automatic 
Document Processing, pages 323-323, New Jersey, 1971: 

where Q is the ideal query, m and 02 are the numbers of positive and negative 
images in the query respectively, and R| and Si are the features of the positive 
and negative Images respectively, a and p determine the relative weighting of 
the positive and negative examples. The values a = 0.65 and p = 0.35, which 
are used for some text-retrieval systems are used (see MQIIer ef a/.). 

[0018] Since the system selects negative example Images 

automatically, a drawback of systems from the first category, is that using 
inappropriate images can destroy the query. Indeed, If the system chooses as 
negative example some Images which should rather be considered as positive 
example, then the relevant features of these images will be discarded, and this 
will mislead the retrieval process. 

[0019] Vasconcelos a/. In Teaming from User Feedback in 

Image Retrieval Systems." in Neural infomiation Processing Systems 12, 
Denver, Colorado, 1999 disclose a content-based image retrieval methods 
Involving negative example from the second category. IVIore specifically, they 
propose a Bayesian model for image retrieval, operating on the assumption 
that the database is constituted of many Image classes. When p^rfomiing 
retrieval, image classes that assign a high membership probability to positive 
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example images are supported, and image classes that assign a high 
membership probability to negative example images are penalized. It is to be 
noted that the authors consider that the positive and the negative examples 
have the same relative Importance. A drawback of the method and system 
proposed by Vasconcelos is that it doesn't perfomi any kind of feature 
weighting of selection. Indeed, it is well known that the importance of features 
varies from one user to the other and even from one moment to another for the 
same user. However, this system considers that all features have the same 
Importance. 

[0020] PIcard a/, in "Interactive Learning Using a 'Society of 

Models' from the IEEE Conference on Computer Vision and Pattem 
Recognition, pages 447--452, San Francisco, 1996., and in "Modeling user 
subjectivity in image libraries". Technical Report No. 382, MIT Media Lab 
Perceptual Computing, 1996, proposed methods Involving searching for the set 
of images similar to positive example, then searching for the set of Images 
similar to negative example; and finally manipulating the two sets in order to 
obtain the set of images to be returned to the user. 

[0021] More specifically, Picard et al. teach the organization of 

database images into many hierarchical trees according to individual features 
such as color and texture. When the user submits a query, comparison using 
each of the trees are performed, then the resulting sets are combined by 
choosing the image sets which most efficiently describe positive example, with 
the condition that these sets don't describe negative example well. 

[0022] Belkin et al. In Rutgers' TREC-6 Interactive track experience, 

from the 6th Text Retrieval Conference, pages 597-610, Galtherburg, USA, 
1998 use a Bayesian probabilistic model in which they assume that the relevant 
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features of positive example are good, wliether or not they are relevant to 
negative example. Their interpretation of negative example is that the context in 
which positive example appears Is inappropriate to the searcher's problem. 
They propose to Increase the (positive) weight of the relevant features of 
positive example (irrespective of their appearance In negative example); and to 
enhance (with negative weights) the relevant features of negative example 
which don't appear in positive example. 

[0023] Belkin et aL consider the negative example at the feature 

level. They try to identify and enhance the features which help to retrieve 
images that are at the same time similar to positive example but not similar to 
negative example. However, enhancing important features of positive example 
which also appear in negative example can mislead the retrieval process, as 
wjll be discusised herelnbelow. 

[0024] Finally, Nastar et aL in "Relevance Feedback and Category 

Search in Image Databases." from the IEEE International Conference on 
Multimedia Computing and Systems, pages 512-517, Florence, Italy. 1999, 
and in "Efficient Query Refinement for Image Retrieval." from the IEEE 
Conference on Computer Vision and Pattern Recognition, pages 547-552, 
Santa Barbara, 1998, consider an image database made up of relevant 
images, among which the user chooses positive example, and non-relevant 
Images, among which the user chooses negative example. A probabilistic 
model Is used to estimate the distribution of relevant images and to 
simultaneously minimize the probability of retrieving non-relevant images. A 
drawback of such a model is Its interpretation of negative example, and how It 
confuses between negative example Images and non-relevant Images. In a real 
database, most Images In general are inrelevant to a given query; however, few 
of them can be used as negative examples without destroying this query.. 
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OBJECTS OF THE INVENTION 

[0025] An object of the present Invention Is therefore to provide 

Improved content-based Image retrieval using positive and negative examples. 

SUMMARY OF THE INVENTION 

[0026] A content-based method for retrieving data files among a set 

of database flies according to the present invention generally aims at defining a 
retrieval scenario where the user can select positive example images, negative 
example images, and their respective degrees of relevance. This allows first to 
reduce the heterogeneity of the dataset on the basis of the positive example, 
then to refine the results on the basis of the negative example. 

[0027] More specifically. In accordance with a first aspect of the 

present invention, there is provided a content-based method for retrieving data 
files among a set of database files comprising: providing positive and negative 
examples of data files; the positive example Including at least one relevant 
feature; providing at least one discriminating feature in at least one of the 
positive and negative examples allowing to differentiate between the positive 
and negative examples; for each database file in the set of database files, 
computing a relevance score based on a similarity of the each database file to 
the positive example considering the at least one relevant feature; creating a 
list of relevant files comprising the Nbl files having the highest similarity score 
among the set of database files; Nbl being a predetermined number; for each 
relevant file in the list of relevant files, computing a discrimination score based 
on a similarity of the each relevant file to the positive example considering the 
at least one discriminating feature and on a dissimilarity of the each relevant file 
to the negative example considering the at least one discriminating feature; and 
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selecting the Nb2 files having the highest discrimination score among the list of 
relevant files; Nb2 being a preidetermined number. 

[0028] In accordance with a second aspect of the present invention, 

there is provided a content-based method for retrieving images among a set of 
database images comprising: providing positive and negative example images; 
the positive example image including at least one relevant feature; providing at 
least one discriminating feature In at least one of the positive and negative 
examples allowing to differentiate between the positive and negative example 
images; for each database image in the set of database images, computing a 
relevance score based on a similarity of the each database image to the 
positive example image considering the at least one relevant feature; creating a 
list of relevant images comprising the Nb1 Images having the highest relevance 
score among the set of database Images; Nb1 being a predetemnined number; 
for each relevant image in the list of relevant images, computing a 
discrimination score based on a similarity of the each relevant Image to the 
positive example Image considering the at least one discriminating feature and 
on a dissimilarity of the each relevant innage to the negative example Image 
considering the at least one discriminating feature; and selecting the Nb2 
images having the highest discrimination score among the list of relevant 
images; Nb2 being a predetermined number. 

[0029] In accordance with a third aspect of the present invention, 

there is provided a content-based method for retrieving images among a set of 
database images, the method comprising: providing positive and negative 
example images; the positive example Image Including at least one relevant 
feature; restricting the set of database Images to a subset of images selected 
among the database images; the images in the subset of images being 
selected according to their similarity with the positive example based on tiie at 
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least one relevant feature; retrieving images in the subset of images according 
to tlieir similarity with the positive example based on the at least one relevant 
feature and according to their dissimilarity with the negative example based on 
at least one discriminating feature between the positive and negative examples; 
whereby, the images retrieved among the database images conresponding to 
images similar to the positive example and dissimilar to the negative example. 

[0030] A content-based image retrieval method according to the 

present invention renders unnecessary the computation of the ideal query since 
it allows to automatically integrate what the user is looking for into similarity 
measures without the need to identify any Ideal point. 

[0031] In accordance to a fourth aspect of the present invention, 

there is provided a content-based system for retrieving images among a set of 
database images comprising: means for providing positive and negative 
example images; the positive example image including at least one relevant 
feature; means for providing at least one discriminating feature in at least one 
of the positive and negative examples allowing to differentiate between the 
positive and negative example images; means for computing, for each 
database image in the set of database images, a relevance score based on a 
similarity of the each database image to the positive example image 
considering the at least one relevant feature; means for creating a list of 
relevant images comprising the Nbi images having the highest similarity score 
among the set of database Images; Nbi being a predetermined number; means 
for computing, for each relevant Image in the list of relevant images, a 
discrimination score based on a similarity of the each relevant Image to the 
positive example image considering the at least one discriminating feature and 
on a dissimilarity of the each relevant image to the negative example image 
considering the at least one discriminating feature; and means for selecting the 




wo 2004/015589 ^BPCT/CAlOOa/OOlZlS 



14 



Nb2 images having the liigliest discrimination score among the list of relevant 
images; Nbz being a predetermined number. 

[0032] In accordance to a fifth aspect of the present invention, there 

is provided an apparatus for retrieving images among a set of database 
images, the apparatus comprising: an Interface adapted to receive positive and 
negative example images; the positive example image including at least one 
relevant feature; a restriction component operable to restrict the set of 
database images to a subset of images selected among the database images; 
the images in the subset of images being selected according to their similarity 
with the positive example based on the at least one relevant feature; a retrieval 
component operable to retrieve images in the subset of images according to 
their similarity with the positive example based on the at least one relevant 
feature and according to their dissimilarity with the negative example based on 
at least one discriminating feature between the positive and negative examples; 
whereby, the images retrieved among the database images correspond to 
images similar to the positive example and dissimilar to the negative example. 

[0033] Finally, in accordance to a sixth aspect of the present 

invention, there is provided a computer readable memory comprising content- 
based image retrieval logic for retrieving images among a set of database 
inriages, the content-based image retrieval logic comprising: image reception 
logic operable to receive positive and negative example images; the positive 
example image including at least one relevant feature; restriction logic operable 
to restrict the set of database images to a subset of images selected among the 
database images; the images in the subset of images being selected according 
to their similarity with the positive example based on the at least one relevant 
feature; and retrieval logic operable to retrieve images in the subset of images 
according to their similarity with the positive example based on the at least one 
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relevant feature and according to their dissimilarity with the negative example 
based on at least one discriminating feature between the positive and negative 
examples; whereby, the images retrieved among the database images 
correspond to images similar to the positive example and dissimilar to the 
negative example. 

[0034] Other objects, advantages and features of the present 

invention will become more apparent upon reading the following non restrictive 
description of preferred embodiments thereof, given by way of example only 
with reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0035] In the appended drawings: 

[0036] Figure 1 is a flowchart illustrating a content-based image 

retrieval method ac(X}rding to an illustrative embodiment of the present 
invention: 

[0037] Figure 2 is a graph illustrating precision-scope curves for two 

cases: negative example in two steps according to the method of Figure 1 and 
negative example, in one step according to the prior art; 

[0038] Figure 3 is a computer screenshot of a graphical interface 

displaying sample images related to different subjects and emphasizing 
different features; 
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[0039] Figure 4 is a computer screenshot of a query screen from a 

user-interface allowing a person to characterized example images accorcling to 
the method of Figure 1; 

[004O] Figure 5 is a schematic view illustrating the decomposition of 

the HIS color space into a set of subspaces and the computation of each 
subs pace's histogram; 

[0041] Figure 6 is a graph illustrating a positive average, a negative 

average, and the resulting overall query average; 

[0042] Figure 7 is a graph illustrating the minimization of the global 

dispersion leading to neglect the relevant features of negative example; 

[0043] Figure 8, which is labeled "Prior Art", is a graph illustrating the 

minimization of the dispersion of positive example, the minimization of negative 
example and the minimization of the distinction between them according to a 
method from the prior art; 

[0044] Figure 9 is a screenshot illustrating the result following step 

106 from the method of Figure 2; 

[0045] Figure 10 Is a screenshot illustrating the result following step 

112 from the method of Figure 2; 

[0046] Figure 1 1 is a graph illustrating precision-scope curves for 

retrieval with positive example and refinement with negative example; and 
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[0047] Figure 12 is a table siiowing the number of iterations needed 

to locate a given category of images in two cases: using positive example only 
and using both positive and negative examples according to the method of 
Figure 2. 

\ DETAILED DESCRIPTION OF THE INVENTION 

[0048] A content-based image retrieval method according to the 

present invention involves relevance feedback using negative examples. The 
negative examples are considered from the feature point of view, and used to 
identify the most discriminating features according to a user-given query. 

[0049] A content-based image retrieval method according to the 

present invention makes use of decision rules including characteristic rules and 
discrimination rules will now be briefly explained. A characteristic rule of a set is 
an assertion which characterizes a concept satisfied by all or most of the 
members of this set. For example, the symptoms of a specific disease can be 
summarized by a characteristic rule. A discrimination rule is an assertion which 
discriminates a concept of the target set from the rest of the database. For 
example, to distinguish one disease from others, a discrimination rule should 
summarize the symptoms that discriminate this disease from others. 

[0050] in applying a content-based image retrieval method 

according to the present invention, it is assumed that positive and negative 
examples possess some relevant features that are discriminant, i.e., relevant to 
either positive or negative example or to both but whose values are not the 
same in positive and in negative examples. In other words, the case in which 
the relevant features of positive example are the same as those of negative 
example, with similar values is excluded. Such a case would yield an 
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ambiguous query. A system implementing a content-based image retrieval 

I 

method according to the present invention is programmed to reject such a case 
and to prompt and allow the user to specify new relevant features. 

[0051] To Implement the above described principle, characteristic 

rules may first be extracted from positive example images by the identification 
of their relevant features. More importance should then be given to such 
features in the retrieval process and images enhancing them should be 
retrieved. Secondly, discrimination rules can be extracted from the difference 
between positive example and negative example. Relevant features whose 
values are not common to positive and negative examples are good 
discriminators, and hence must be given more importance; conversely, 
common features are not good discriminators, and must be penalized, 
l-lowever, applying this principle in this manner, may render misleading the 
retrieval process by neglecting certain relevant features of positive and 
negative examples, as explained below. 

[0052] Before describing in details a content-ba^ed image retrieval 

method according to the present invention, which would solve the problem 
presented hereinabove, the concept of relevant feature will be define in more 
detail. A given feature is considered relevant if it helps retrieving the images 
being sought. This will depend on two factors. 

[0053] First, the relevance can be considered with respect to the 

query. A feature relevant to the query is a feature which is salient in the 
majority of the query images. A feature to be considered is a feature whose 
values are concentrated in the query images, and which discriminates well 
between positive and negative examples, as relevant to the query. 
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[0054] Second, the relevance of a feature can be considered with 

respect to the database. If a given feature's values are almost the same for the 
majority of the database Images, then this feature Is considered to be not 
relevant since It doesn't allow to distinguish the sought images from the others; 
and vice versa. To illustrate this, consider a database in which each Image 
contains an object with a circular shape, but where the color of the object 
differs from one image to another. In such a database, the shape feature is not 
interesting for retrieval since it doesn't allow to distinguish between desired and 
undesired irriages; however, the color feature is interesting. In other words, a 
feature in term of which the database is homogeneous is considered not 
relevant for retrieval; whereas, a feature in term of which the database is 
heterogeneous is considered relevant. 

[0055] In the following, the consequences of neglepting features 

whose values are common to both positive and negative examples lis analyzed. 
In fact, this depends on the nature of the database. If the database is 
homogeneous In terms of such features, then neglecting them will not be 
detrimental since they are not relevant to the database. On the other hand, If 
the database is heterogeneous in terms of these features, then neglecting them 
will lead the system to retrieve many undesired Images and to miss many 
desired images. 

[0056] From the above. It is clear that common features should be 

considered to develop a solution that works for any query. However, in some 
cases, there are not enough common features to be considered alone at a 
given moment; they must rather be considered together with other features. 
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[0057] Turning now to Figure 1 of the appended drawings, a 

content-based image retrieval method 100 according to a first illustrative 
embodiment of the present invention is illustrated. 

[0058] Generally stated the method 100 consists in performing the 

following steps: 

102 - providing a set of database images; 

104 - providing positive and negative example images; 

106 - for each database image, computing a relevance score 
based on a similarity of the database image to the positive example 
image considering relevant features; 

108 - creating a list of relevant images comprising the Nbi 
images having the highest relevance score among the set of database 
images; 

110 - providing discriminating features allowing to 
differentiate between th6 positive and negative example images; 

112 - for each relevant image in the list of relevant images, 
computing a discrimination score based on the similarity of tlie relevant 
image with the positive example image considering the discriminating 
features and on a dissimilarity of the relevant image with to the negative 
example image considering the discriminating features; and 

114 - selecting the Nb2 images having the highest 
discrimination score among the list of relevant images. 

[0059] It can be useful to described a content-based image retrieval 

method according to the present invention as including two general steps. In 
the following, we will refer to the steps of the method 100 using referral 
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numbers and we will refer to the more general steps using the expressions: first 
and secx}nd general steps. 

[0060]. The first general step allows to reduce the heterogeneity of 

the set of images participating in the retrieval by restricting it to a more 
homogeneous subset accoixiing to positive example relevant features (and thus 
according to common features also). In this first general step, we enhance ail 
the relevant features of positive example. We rank the database images 
according to their resemblance to positive example and then retain only the Nbi 
top-ranked Images, where Nbi is a predetermined number. 

[0061] Only images retained in the first general step will participate 

in the refinement perfonned in the second general step, where we enhance the 
discrimination features, i.e., those whose values are not common to positive 
and negative examples. In this second general step we rank the candidate 
Images according to their similarity to positive example and dissimilarity to 
negative example, and return to the user only the Nba (Nb2 < Nbi) top-ranked 
images. Hence, even if the common features are neglected in the second 
general step, this will not mislead the retrieval since they were considered in 
the first general step. As will be presented hereinbelow in more detail, we 
confirmed experimentally, using a retrieval system implementing the present 
method, the importance of processing queries with negative example in two 
steps. 

[0062] Figure 2 compares the curves precision-scope for the two 

techniques: negative example queries processed in two general steps 
according to a content-based image retrieval according to the present invention 
versus negative example queries processed in a unique step (in which both 
positive and negative examples are considered and all images in the database 
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participate in retrieval) according to methods from the prior art. The ordinate 
"Precision" represents the average of relevance of retrieved images, and 
"scope" is the number of retrieved images. It is clear fronn Figure 1 that when 
queries containing negative example are considered in one step, the precision 
of retrieval decreases quickly with the number of retrieved images. 

[0063] Before describing each of the steps 102-114 of the method 

100, some special cases are important and merit to be mentioned to show that 
the proposed image retrieval method functions as well. These cases emerge 
when all the discrimination features come from positive example only or from 
negative example only. Indeed, if the relevant features of positive example are 
strictly included In those of negative example and with common values, then 
applying the proposed principle leads, in the general first step, to enhance the 
relevant features of positive example (which are the same as the common 
features) and to retain images looking like it. Then, in the second general step, 
to enhance the rest of the negative example relevant features and to discard 
images near to it. On the other hand, if the relevant features of negative 
example are strictly included in those of positive example and with common 
values, then applying the proposed principle leads, in the first general step, to 
enhance the relevant features positive example (which include those of 
negative example) and to retain images looking like the positive example. 
Then, in the second general step, to enhance only those features relevant to 
positive but not to negative example and to re-rank the images according to 
these features essentially. 

[0064] The following will explained how the content base Image 

retrieval method 100 may allow a user to compose a query using negative 
example only. 
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[0065] First, we note that, for a given query, the number of non- 

relevant images is usually much higher than the number of relevant images. In 
other words, if we know what someone doesn't want, this doesn't Inform us 
sufficiently about what the user wants. For example, if the user gives an image 
of a car as negative example without giving any positive example, then we 
cannot know whether the user is looking for images of buildings, animals, 
persons or other things. Nevertheless, negative example can be used alone in 
some cases, for instance, to eliminate a subset from a database, for example, 
when a database contains, in addition to images the user agrees with, other 
images that the user's culture doesn't tolerate, e.g. nudity images for some 
persons. In such a case, the user can first eliminate the undesired images by 
using some of them as negative example; then the user can navigate in, or 
retrieve from the rest of the database. Concerning the retrieval method, the 
negative-example-only query will be considered as a positive example query. 
I.e., the system first searches for images that resemble negative example. 
Then, when the resulting Images (images that the user wants to discard) are 
retrieved, the system returns to the user the rest of the database rather these 
images. 

[0066] Each of the steps 102-114 of the method 100 will now be 

described in more detail. 

[0067] In step 102, a set of database images is provided to or by a 

user, among the set of images possibly including images that the user wants to 
retrieve. 

[0068] Then, in step 104, positive and negative example Images are 

provided through interaction between the user and the system implementing 
the method 100. Of course, the person seeking images having specific 
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features can alternatively select the example images manually. In that case, 
the selected images are digitized aftenvards. 

[0069] The user Interaction alms to achieve two main objectives. 

First, to be able to combine the query Images together with their respective 
degrees of relevance In order to identify what the user is looking for; and to 
integrate this Information in similarity measures. Second, to weight each 
predetermined feature and its components according to its relevance to the 
query and the discrimination power it can provide. 

[0070] Figure 3 illustrates a graphical Interface displaying nine 

sample images related to different subjects and emphasizing different features. 
The graphical Interface Is programmed so as to allow a user to choose 
additional Images from the database before fonnulating the query. To select an 
image as an example image (or query image), the user may click on the 
"Selecr button. The system displays a dialog box allowing the user to specify a 
degree of relevance (see Figure 4). The user-interface illustrated in Figure 4 
allows a person to characterize selected example Images. 

[0071] For each selected images, the possible relevance degrees 

are 

• Very similar: con^esponds to the relevance value 2 for a 
positive example image; 

• Similar: corresponds to the relevance value 1 for a positive 
example image; 

• Doesnt matter the image will not participate in the query; 
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• Different: corresponds to the relevance value 1 for a negative 
example image; or 

• Very different: corresponds to tlie relevance value 2 for a 
negative example image. 

[0O72] Of course, the relevancy of each image can be characterized 

with more or less finesse. 

[0O73] Before explaining in more detail the fomriulation of relevance 

feedback, an example of image model and similarity measure will be described. 
Of course, another Image model can altematively be used. 

[0O74] To represent images, the hierarchical model proposed by Rui 

e( al. is used. According to this model, each image, either in the query or in the 
database, is represented by a set of I features, each of which is a real vector of 
many components. It has been found that this image model ensures a good 
modeling of both images and image features, and a reduction in the 
computation time. According to this hierarchical two-level image model, a 
distance metric for each level is selected. For feature level, a generalized 
Euclidean distance function is chosen, as in Ishikawa etal. If 3c„and 3c,2.are the 

i**^ feature vectors of the images xi and xa respectively, then the distance at this 
feature level is 

A(^il, ^£2) = {Xix — Xi2)'^Wi(^ii — Xi2) 

(4) 

where Wi is a symmetric matrix that allows us to define the generalized 
ellipsoid distance D|. 
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[0075] The choice of this distance metric allows not only to weight 

each feature's component but also to transform the initial feature space into a 
space that better models the user's needs and specificities. The global distance 
between two images xi and X2 is linear and is given by 

I 

D{xu 3:2) = ^ Ui{xii - X2ifWi{Xii —S^2i) 

«=i 

(5) 

where ui is the global weight assigned to the i*^ .feature. 

[0076] Each image, either in the database or in the query, is 

represented by a set of 27 feature vectors, computed as follows: First, every 
pixel in the image is mapped to a point in the three-dimensional (3D) HSI space 
(Figure 5). This operation consists of computing, for every triple [H.S.I], the 
number of pixels having the values Hue = H, Saturation = S and Intensity = I. 
This yields a 3D color histogram that takes up a lot of space and having zeros 
for most of its values. For example, an image with HSI values ranging between 
0 and 255, would yield a histogram containing 256^ cells, most of which not 
corresponding to any pixel. 

[0077] To reduce the histogram's size, many solutions are possible, 

such as the spatial repartition of the points of the 3-D histogram, taking into 
account their respective occun-ence frequency, i.e., the number of pixels 
corresponding to each point in the histogram. However, since the method 100 
does not aim at finding the best visual features, a compromise consists in 
partitioning the space by subdividing the axes H, S and I into three equal 
intervals each. This gives 3^ = 27 subspaces, as shown in Figure 5. Each 
subspace constitutes a feature, and its corresponding vector is computed as 
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follows. The subspace Is subdivided Into 2^ = 8 sub-subspaces. The sum of the 
elements of each sub-subspace is computed and the result Is stored In the 
corresponding cell of the feature vector 

[0078] Altematlvely, the innages can be represented using other 

models. 

[0079] In step 106, a relevance score Is computed for each 

database image based on the similarity of the image to the positive example 
Image considering the relevant feature. 

[0080] Considering that the user constructs a query composed of Ni 

positive example images and their respective relevance degrees tt^ for n = 

1 Ni, as well as Na negative example images and their respective relevance 

degrees tt^ for n = 1,....N2. (It should be noted that is not the square of ^„ ; 
2 is an index designating the negative example). 

[0081] Only the positive examples are considered in step 106. Each 

relevance feature and its components is enhanced according to its relevance to 
the positive example. This can be done by introducing the optimal parameters 
U| and W| which minimize Jpositive. the global dispersion of positive example, 
given in Equation (6). 

I Ni 

JposiUve = Y1'^YI "^niP^ni ~ " ) 

i=l n=l (6j 

Where x] is the weighted average of positive example (see Figure 6), given by 



wo 2004/015589 




PCT/CA2003/001215 



■in=l ':n 



(7) 

[0082] An image retrieval method according to the present invention 

allows to give more weight to features and feature components for which the 
positive example images are close to each other In the feature space. An 
informal justification is that if the variance of query images is high along a given 
axis, any value on this axis is apparently acceptable to the user, and therefore 
this axis should be given a low weight, and vice versa. 



[00^3] In step 108, the database images are ranked In increasing 

order accxjrding to a relevance score based on a similarity of each database 
image to the positive example image considering the relevance features 

[0084] More specifically a distance from the positive example 

average and the Nbi top-ranked Images is computed are kept for the next 
steps. This distance is given by Equation (8). 

'=1 (8) 
[0085] If the query contains only negative example images, then the 

system proceeds initially by a similar procedure, but considering the negative 
example rather than the positive example. This means that the system 
computes the ideal parameters which minimize the dispereion of negative 
example images, ranks the Images in Increasing order according to their 
distance from the negative example average, then returns to the user the last- 
ranked innages. If the query contains both positive and negative examples, then 
the system perfomns the two steps of retrieval. The parameter computation and 
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the distance function used In the first step are the same as in the case of a 
positive-example-only query. 

[0086] In the second general step, both positive and negative 

example images are considered, and the refinement concems the images 
retained in the first general step and more specifically in step. 108. 

[0O87] First Jgiobai. the global dispersion of the query, including 

positive and negative example images is defined: 

/ 2 Nk 

i=l fc=l n=l (9) 

Where k = 1 for positive example and k = 2 for negative example, and where q. , 

given In Equation (10), is the weighted average of all query images for the i* 
feature (see Figure 7). 

a< _ ^k=l ^n=l "w^'m 

2Lik=\ 2^n=l'^n i^O) 

[0O88] In Rui et al. (2), it is proposed to allocate negative degrees of 

relevance to negative example images and to compute the parameters which 
minimize the same expression of Equation (9). The consequences of such an 
approach, which is not adopted in a content-based image retrieval method 
according to the present invention, will now be considered in order to emphasis 
the differences such an approach and the one used in the method 100. If 
positive example are considered separately from negative example In Equation 
(9), then: 
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(11) 

[0089] Rui e* a/. (2) choose < > 0 for n = 1 , . . ..Ni and tvI < 0 for n = 

1 N2 , yielding: 

Jgiobal = E ^ E^ - qii^Wi{SU - Si) - E^« E knK^ni - ®rW"i(^ni - Qi) 

(12) 

[0090] where \ jtl \ designates the absolute value of tvI. Equation 

(12) shows that the global dispersion Jgiobai is the dispersion of positive 
example minus the dispersion of negative example. Hence, by minimizing the 
global dispersion, even if Rui et al. (2) move the global query average q (with 
which they compare their images) towards positive example and away from 
negative example, two problems emerge. 

[0091] First, minimizing the global dispersion will lead to minimize 

the dispersion of positive example, but with respect to the global query average 
q rather than the positive example average 5cj . This will not give an optimal 
minimization of the positive example dispersion; and hence, the relevant 
features of positive example will not be given enough importance. 

[0092] Second, minimizing the global dispersion will lead to 

maximize the dispersion of negative example. This implies that they neglect the 
relevant features of negative example. Hence, their retrieval system will not be 
able to discard the undesired images. This is illustrated In Figure 8. 
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[0093] 



The weights ui and Wi are introduced to give more 



importance to the relevant features of either positive or negative example which 
allow to distinguish well between them. In other words, via Ui and W|, weights 
are attributed to features and the feature space is transformed into a new 
space in which positive example images are as dose as possible, negative 
example images are as close as possible, and positive example is as far as 
possible from negative example (see Figure 7). These objectives are translated 
into a rriathematical formulation, by first distinguishing positive example images 
from negative example images in the global dispersion fomnula of Equation (9). 
For each feature i, the weighted average of positive example images xj is 
recalled and the weighted average of negative example images xf 'in Equations 
(13) and (14) respectively is defined. 




(13) 




(14) 



[0094] 



By introducing and into Equation (9), one can rewrite it 



as follows: 



j^iot^i = E"* E E - 1;) + (i* - m)]Wi[{^ni - 



(15) 



[0095] 



Developing Equation (15) gives 
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/ r, 2 iVfc 2 Mjfc 

Joiotai - E^' ( E E 4(5^- - - )) + ( E E - s'ifma^ - g,)) 
+( E E ''^(S? - fif - s?)) + ( E E "Jc^f - - m 

fcslTUsl Jfearlnssl "^J 

(16) 

[0096] It can easily be shown that the second and third parts of 

Equation (1 6) are zero. For example, the second part 

ELi Ti^t <i^ni - ^rw,{f, - ^) = [( e£i ^^i^ni - ^ir)mi^i - ^-)] 

since, according to Equations (13) and (14), 

i:^i<xti - (E^Jii ost = 0 

[0097] Thus, Equation (1 7) can be written as follows: 

= [E E E 4(4* - ^fw^*(«£, - [ E E T^c^ - - m] - 

(17) 

[0098] The first temn "A" expresses the positive example internal 

dispersion, i.e., how close positive example images are to each other, added to 
the negative example internal dispersion, i.e.. how close negative example 
images are to each other. The second temi "R" expresses the distance 
between the two sets, T.e., how far positive example is from negative example. 

[0099] By distinguishing the Intra dispersion "A" from the inter 

dispersion "R", it is now clearer how one can formulate the above-identified 
objectives in a mathematical problem. In fact, one want to compute the model 
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parameters, namely ui and W|, which minimize the intra dispersion "A" and 
maximize the inter dispersion "R". Several combinations of A and R are 
possible. 

[00100] The parameters which minimize the ratio — , assuming that 

R 

i??to will be computed. In the case of R = 0, the positive example and the 
negative example are not distinguishable and the query is ambiguous. In such 
case, the query is rejected and the user is asked to formulate a new one. 
Furthermore, to avoid numerical stability problems, the following two 

constraints are introduced: Y^^^— = lar)6 det(Wi)=1 for all i=1,...,l. By using 

Lagrange multipliers, the optimal parameters ui and W| must minimize the 
quantity L given in Equation (18). 



^ ^i=i«» ^ i=i (18) 



where 



I ' 2 JVfo 

i=l A:=ln=l (19) 



and 



^ = E «i E ^'(^ - ^rWii^t - gi) 

i=i *=i (20) 

^'denotes the sum of positive example relevance degrees, i.e., =2^11 ^« 
and denotes the sum of negative example relevance degrees, i.e., 
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[00101] The optimization problem in order to obtain tlie optimai 

parameters U] and Wi will now be resolved. 

[00102] It Is to be noted first that the relative importance of positive 

and negative examples are to be determined, i.e., ^' with respect to n'^ . Some 
image retrieval systems, such as the one described by MUller a/, adopt the 
values used by certain text retrieval systems which are 0.65 for positive 
example and 0.35 for negative example. Other systems such as the one 
described by Vasconcelos ef a/, assume that positive example and negative 
example have the same Importance. In the method 100, the latter choice is 
adopted because It allows some simplifications In the derivation of the problem. 
Furthermore, all the user-given relevance degrees are nonmallzed so that 

[001 03] To obtain the optimal solution for Wi, the partial derivative of L 
with respect to w,^ for r,s=1 Hi, Is taken where Hi is the dimension of the l"" 

feature and w,^ Is the rs* element of Wi. I.e., Jf, =[w,^], yielding 



where 

dA ^ 

W«^r» fe=l»=:l (22) 

and 

^^^rs k=l (23) 
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[00104] . Before computing It Is to be noted that 

det(JF,) = 2;"',(-l)'"v»';„ det(W,.J, where det(FI^Jis the rs*^ minor of W, obtained 
by eliminating the r^ row and the s*^ column of det(FF,) . Hence, 



(24) 



By substituting Equations (19). (20) and (21) in (1 8), we obtain 

a — = 0<^ 



■ 2 a 

(25) 



[001 05] Now consider the matrix W~' = [ w"* ] , the inverse matrix of W| 

(provided that Wj is invertible). To obtain the value of each component w"* , the 
determinant method for matrix inversion is used to obtain 



Knowing that det(Wi)=1 yields 



*r« 

(26) 
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[00106] In Equation (26), det(B^_) is replaced by Its value from 

Equation (25) to obtain 



2 yVfc 2 

(27) 



where 

/y =: ^^^^ 

[00107] Equation (27) can also be written in matrix form as 

y (28) 

where Q is the matrix [c,^] such tiiat 

2 iV& .2 

Aj=1 n=:l 

(29) 

[00108] The value of y will now be computed independently firom X 

which is an unknown parameter. Equation (28) can be written as follows: 

but since det(FF;"^) = l, then x = (det(C;))'^'C,'^ Finally, the optimal solution for 
Wi is given by Equation (30) 
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where the components of C| are given by Equation (29). 

[00109] In the following, the effect of the dispersion of positive and 

negative examples on the components of Wj will be considered. Firet. Equation 
(29) can be rewritten in a matrix form, as follows: 

Ci = RCovai — ACovTi 

(31) 

where Covai is the sum of intra covariance matrices for the i"" feature, i.e., 
Cova, = [cov a,_^ ]such that 

and Covri is the Inter covariance matrix for the i* feature, i.e., Cowj =[covr^^] 
such that 

[00110] • Now, considering Equation (31), where the values of "A" and 

"R" are set since they concern ali the features. If the Intra dispersion Is high 
relative to the inter dispersion, and hence the elements of Covai are Important 
relative to the elerhents of Covn then, according to Equation (31), the values of 
the components of C| will be important. But since 1^, = K^,"* (Equation(30)), it 
follows that the values of w,^ will be small; and consequently, the I*' feature's 
components will be given low weights. On the other hand. If the intra dispersion 
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is low relative to tlie inter dispersion for the i*'' feature, by a similar line of 
reasoning, one can see that this feature's components will be given high 
weights. This behavior of Wi fulfills the objective of enhancing discriminant 
features against other ones. 

[O0111] Taking the partial derivative of L with respect to ui allows to 

obtain the optimal solution for U|. 



dui u? 



(32) 



where 



k=ln=l (33) 



and 



OR 2 

dui 



ife=i 



(34) 



[O01 12] By substituting Equations (33) and (34) in (32), we obtain 

' ^ = o*Mi[i:x:^(^^*-^m(5^*-sf)] -^[E**(s?-«)^m(5Nfli)] + ^=o 

(35) 

[O01 13] Both sides of Equation (35) are multiplied by ui, to obtain: 



^ (36) 
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where 

(37) 

[00114] Now, to get rid of the unknown parameter A,, a relation, 

Independent of A , between Ui and any Uj Is sought. First A can be computed 
directly from Equation (36) as follows: 



X = -M vi 

^ (38) 

[00115] Second, taking the sum on i of Equation (36) gives 

Xl,«,/,+^'X>,;7-=0. but since i;t,:;^=l. then ^.ujfj+JlR^ =0. It 

follows that 



Uj — 



-R^ (39) 
[001 16] Equations (32) and (33) imply that for every feature i 



I 

^■=1 (40) 
[001 1 7] It follows from Equation (40) that f^uf = f^ul = ... = f,uf = fjuj . 



[00118] 



Hence, 
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[fi 

V J3 (41) 

[00119] Finally, to obtain the optimal solution of U|, iij is replaced in 

Equation (40) by its value fronn Equation (41 ), yielding: 



j=i V ■'J j=i 

^ — 

V/i (42) 

[00120] The optimal solution for U| is given by Equation (42), where fj 

is defined by Equation (37). 

[00121] The influence of the dispersion of positive and negative 

examples on the value of each ui will now be considered First, fi can be written 
in Equation (37) as 



Si = RFtti - AFn 

where 

h=zl n=l 

and 



(43) 



(44) 



fc=l (45) 
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[00122] It is assumed that A and R have constant values since they 

depend on all the features. If, for the i* feature, the Intra dispersion Is high 
relative to the Inter dispersion, then the quantity Fai will gain In importance 
relative to the quantity Fn. According to Ec^uatlon (43), this will increase the 
value of f|. Moreover, Equation (42) shows that when f| increases, ui decreases; 
and hence, the I**" feature will be given a low weight. Conversely, if, for the I*' 
feature, the intra dispersion Is low relative to the Inter dispersion, then, by a 
similar line of reasoning, we find that the i*^ feature will be given a high weight. 
Therefore, the optimal value that is found for Uj fulfills the objective of 
enhancing the relevant discriminant features against others. 

[00123] In brief, the Input to step 112 consists of positive example 

Images, negative example images and their respective relevance degrees. A 
partial result of step 112 includes the optimal parameters Wi and ui. These 
parameters are computed according to Equations (30) and (42), respectively. 
The computation of these parameters requires the computation of x}, f? , f|, 
A and R according to Equations (13), (14), (10), (37), (19) and (20), 
respectively. The algorithm is iterative since the computation of W| and ui 
depends on A and R, and the computation of A and R depends on W| and ut. 
The fixed point method is used to perform the computation of W| and U|. An 
initialization step Is required, in which we adopt the following values: 



[00124] 



- Wi is Initialized with the diagonal matrix 
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1 



0 \ 



where 



\ 



0 



O'ir = 



2 Nk 

\ fc=:l7l=l 



is the standard deviation of the r*^ component of the i**^ feature computed for the 
full set of query images. 



[00125] 

by 



- The parameter U) Is initialized with a kind of dispersion given 



where 



Ui — 



ELi E^ii -^jxj, - x^i^Wii^^i - g?) 



[00126] The computation of Wi requires the inversion of the matrix C|. 

However, in the case of (Ni+N2)<H|, Q is not invertible. ishikawa et al. suggest 
proceeding by singular value decomposition (SVD) to obtain the pseudo 
inverse matrix. However, this solution doesnt give a satisfactory result. 
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especially when (Ni+N2)is far less than H| as pointed out by Rui et al., who 
propose, in the case of a singular matrix, to replace Wi by a diagonal matrix 

whose elements are the inverse of the standard deviation, i.e., w. =— if r = s 



and vK^=0 elsewhere. 



[00127] in step 112, W| is replaced by a diagonal matrix whose 

elements are the inverse of the diagonal elements of the matrix Q, i.e., 



ill 



0 



0 



where = — and c,^ can be obtained by setting r = s in Equation (26). 



[00128] In step 114, the relevant images obtained in step 108 are 

ranked according to a discriminating score based on their closeness to the 
positive example and their famess from the negative example. The comparison 
function is given by Equation (44). Finally, the system retums the .Nb2 top- 
ranked images to the user. 



(46) 



PCT/CA2003/001215 
44 



Experimental results and performance evaluation 

[00129] Tests were performed on 10 000 Images from The 

Pennsylvania State University images database, which is .described by J. Li, 
J.Z. Wang and G. Wiederhold in both "\RM: Integrated region matching for 
image retrieval." From the 2000 ACIVl Multimedia Conference, pages 147-156, 
San Jose, USA, 2000. and "SIMPLIcity: Semantics-sensitive Integrated 
Matching for Picture Libraries." from IEEE Transactions on Pattern Analysis 
and Machine Intelligence, 23(9):947-963, 2001. This database contains 
images related to different subjects, emphasizing different features, and taken 
under different Illumination conditions. For each image, the set of features Is 
computed as explained above; Many tests were performed for retrieval and 
refinement. Even when positive and negative examples are not readily 
distinguishable, the method according to the present invention succeeded in 
Identifying discrimination features and sorting the resulting Images according to 
these features. 

[00130] Figure 9 shows an example of retrieval with positive example 

only. Figure 10 shows and example of retrieval with positive and negative 
examples. 

[00131] In the first example, two Images participated in the query as 

positive example. Both of these images contain a green tree under the blue sky 
(5095.ppm and 5118.ppm). Figure 9 shows the top nine returned images. It is 
to be noted that the two query images are returned in the top positions. There 
are also some other images containing trees under the sky, but including noise 
consisting of three images of a brown bird on a green tree under the blue sky 
(5523.ppm, 5522.ppm, 5521. ppm). At the same time, there have been miss, 
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because the database contains other images (not shown) of trees under the 
sky that have not been retrieved. 

[00132] According to the second example, a refinement has been 

applied to the results of the first example. Hence, we use the same images 
(5095.ppm and 5118.ppm) as positive example, while an image of a bird on a 
tree under the sky is chosen as negative example (image 5521 .ppm of Figure 
8). Figure 9 shows that images of birds are discarded (the noise reduced) and 
that more images of trees under the sky are retrieved (the miss decreased). 

Performance evaluation 

[00133] In order to validate the proposed relevance feedback 

technique, a performance evaluation of a retrieval system implementing a 
method according to the present invention has been has been performed. The 
evaluation was based on comparison between the use of positive example only 
and the use of both positive and negative examples. To perform any evaluation 
in the context of image retrieval, two main issues emerge: the acquisition of 
ground truth and the definition of performance criteria. For ground truth, human 
subjects were used: three persons participated in all the experiences described 
hereinbelow. The performance criteria, Precision Pr and Recall Re, described 
by John R. Smith in "Image Retrieval ES/aluation." From the IEEE Workshop on 
Content-based Access of Image and Video Libraries, 1998 were used. 

[00134] In their simplest definition, Precision is the proportion of 

retrieved images that are relevant, i.e., number of retrieved images that are 
relevant on the number of all retrieved images; and Recall is the proportion of 
relevant images that are retrieved, i.e., number of relevant images that are 
retrieved on the number of all relevant images in the database. Smith drew up 
the precision-recall curve Pr=f(Re); however, it has been observed that this 
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measure is less meaningful in tlie context of image retrieval since Recall is 
consistently- low. Furthermore, It is believed that it is often difficult to compute 
Recall, especially when the size of the image database is big; because this 
requires to know, for each query, the number of relevant images in a the whole 
database- Another problem with Recall, is that it depends strongly on the 
choice of the number of images to return to the user. If the number of relevant 
images in the database is bigger than the number of images returned to the 
user, then the recall will be penalized. A more expressive curve which Is the 
precision-scope curve Pr=f(Sc), as described by Huang et aL, "Image Indexing 
using Color Correiogram." From the IEEE Conference on Computer Vision and 
Pattern Recognition, 1997, has been used. Scope Sc is the number of images 
returned to the user, and hence the curve Pr=f(Sc) depicts the precision for 
different values of the number of images returned to the user. Since these 
performance criteria are believed to be well known in the art, they will not be 
described herein in further detail. 

[00135] Two experiences were carried out, each of which aiming to 

measure a given aspect of our model. The first experience aims to measure the 
improvement, with negative example, in the relevance of retrieved images. The 
second experience aims to measure the improvement, with negative example, 
in the number of iterations needed to locate a given category of images. 

First experience 

[00136] As mentioned above, the goal of the first experience is to 

measure the contribution of negative example in the improvement of the 
relevance of retrieved images. Each human subject participating in the 
experience was asked to formulate a query using only positive example and to 
give a goodness score to each retrieved image, then to refine the results using 
negative example and to give a goodness score to each retrieved imiage. The 
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possible scores are 2 if the image is good, 1 if the image is acceptable, and 0 if 
the image is bad. Each subject repeated the experience five times by 
specifying a new query each time. Precision was computed as follows: Pr = the 
sum of degrees of relevance for retrieved Images / the number of retrieved 
images. Figure 1 1 illustrates a comparison between the curves Pr=f(Sc) in the 
two cases: retrieval with positive example and refinement with negative 
example. 

[001371 The experiences shows that, in average, when negative 

example is introduced, the improvement in precision is about 20 %, In fact, the 
improvement varies from one query to another, because it depends on other 
factors such as the choice of a meaningful negative example and the 
constitution of the database. If, for a given query, the database contains a little 
number of relevant images, most of which have been retrieved in the first step, 
then the introduction of negative example or any other technique will not be 
able to bring any notable improvement 

Second experience 

[00138] The second experience aims at measuring the improvement 

in the number of refinement iterations needed to locate a given category of 
images, as well as the role of negatiye example in resolving the page zero 
problem (finding a good image to initiate the retrieval). Each of our human 
subjects was shown a set of images that are relatively similar to each other with 
respect to the color. None of the showed images appear in the set of images 
the subjects can use to formulate the initial query. Each subject is asked to 
locate at least one of the showed images using only positive example, and to 
count the number of iterations; then to restart the experience but using both 
positive and negative examples, and to count the number of iterations. This 
experience was repeated four times and the results are given in Figure 12. S1, 
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S2 and S3 designate respectively the three human subjects who participated in 
the experiments. PE means positive example and NE means negative 
example. Each entry in the table gives the number of iterations needed to 
locate the searched Images. 

[00139] It has been found that when they used both positive and 

negative examples, the subjects succeeded in all the experiences; however, 
when they used only positive example, some of them failed in certain 
experiences to locate any sought image. In Experience 2,2 and Experience 2.4, 
at least one subject was unable to locate any sought image using positive 
example only. This is because, in a given iteration, all the retrieved images fall 
into an undesired category, and the formulation of the next-iteration query using 
any of these images leads to retrieve images belonging to the same category. 
The user can loop indefinitely, but will not be able to escape this situation by 
using positive example only. The second observation is that the use of negative 
example reduces, appreciably the number of iterations. If one computes the 
average number of iterations among the successful experiences (2.1 and 2.3), 
one finds 5.83 when only positive example is used, and 2.33 when both 
positive and negative examples are used. This experience shows clearly the 
role of negative example in mitigating the page zero problem, indeed, after 
having obtaining at least one of the sought images, the user can use it to 
formulate a new query, and hence to retrieve more sought images. 

[001 40] A content-based image retrieval method according to the 

present invention allows to take into account the user's needs and specificities, 
, which can be identified via relevance feedback. It has been shown that the use 
of positive example only isn't always sufficient to determine what the user is 
looking for. This can be seen especially when all the candidate images to 
participate in the query appear in an inappropriate context or contain, in 
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doesn't want to retrieve. 

[00141] It is to be noted that the present model is not limited to Image 

retrieval but can be adapted and applied to any retrieval process with relevance 
feedback. For example, a method according to the present invention can be 
used any process of retrieval such as retrieval of text, sound, arid multimedia. 

[00142] Although the present invention has been described 

hereinabove by way of preferred embodiments thereof, it can be modified, 
without departing from the spirit and nature of the subject invention. 
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WHAT IS CLAIMED IS: 

1 . A content-based method for retrieving data files among a 
set of database files comprising: 

providing positive and negative examples of data files; said 
positive example including at least one relevant feature; 

providing at least one discriminating feature in at least one of said 
positive and negative examples allowing to differentiate between said positive 
and negative examples; 

for each database file in said set of database files, computing a 
relevance score based on a similarity of said each database file to said positive 
example considering said at least one relevant feature; 

creating a list of relevant files comprising the Nbi files having the 
highest similarity score among said set of database files; Nbi being a 
predetermined number; 

for each relevant file in said list of relevant files, computing a 
discrimination score based on a similarity of said each relevant file to said 
positive example considering said at least one discriminating feature and on a 
dissimilarity of said each relevant file to said negative example considering said 
at least one discriminating feature; and 

selecting the Nba files having the highest discrimination score 
among said list of relevant files; Nb2 being a predetermined number. 

2. A content-based method for retrieving images among a set 
of database images comprising: 

providing positive and negative example images; said positive 
example image including at least one relevant feature; 
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providing at least one discriminating feature in at least one of said 
positive and negative examples allowing to differentiate between said positive 
and negative example images; 

for each database image in said set of database images, 
computing a relevance score based on a similarity of said each database 
image to said positive example image considering said at least one relevant 
feature; 

creating a list of relevant images comprising the Nbi images 
having the highest relevance score among said set of database images; Nbi 
being a predetermined number; 

for each relevant image in said list of relevant images, computing 
a discrimination score based on a similarity of said each relevant image to said 
positive example image considering said at least one discriminating feature and 
on a dissimilarity of said each relevant image to said negative example image 
considering said at least one discriminating feature; and 

selecting the Nb2 images having the highest discrimination score 
among said list of relevant images; Nb2 being a predetermined number. 

3. A method as recited in claim 2, wherein said at least one of 
said positive and negative examples being the weighted average of a plurality 
of images. 

4. A method as recited in claim 2, wherein said at least one 
relevant feature includes a number / of relevant features, 

5. A method as recited In claim 4, wherein said positive 
example image being the weighted average x} of Ni positive examples for 
each relevant feature /. 
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6. A method as recited in claim 5, wlierein ^ Is defined by: 
wiierein ttI is a relevance degree for tiie positive example n. 



7. A rrietlnod as recited in claim 6, wherein said at least one 
discriminating feature includes a number / of discriminating features: said 
negative example image being the weighted average of A/2 negative 
examples for each relevant feature /; x? being defined by: 

wherein jri is a relevance degree for the negative example n. 

8. A method as recited in claim 7, wherein tt^ -1 where: 

9. A rnethod as recited in claim 8, wherein ^i=0.5 and 

7^2=0.5 . 

10. A method as recited in claim 2, wherein each of the set of 
database images and of the positive and negative example images is 
represented by a set of image features. 

11. A method as recited in claim 3, wherein each of said set of 
image features being represented by a feature vector. 
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12. A method recited in claim 11, wherein computing a 
relevance score includes computing the distance between said positive 
example image and said each database image; said highest relevance score 
corresponding to the lowest of said distance between said positive example 
image and said each database image. 

13. A method as recited in claim 12, wherein said at least one 
relevant feature includes a number / of relevant features; said positive example 
image is the weighted average ^ of Ni positive examples for each relevant 
feature /; I? being defined by: 

wherein ;r* is a relevance degree for the positive example n; 

said distance between said positive example image and said each database 

image represented by feature vector f„, being defined by: 

i=:l 

wherein Ui is the global weight assigned to the i* relevant feature; and 

Wi is a symmetric matrix that allows defining the generalized 
ellipsoid distance D and weighting components of each of said at least one 
relevant feature; and Ui and W/ minimizing the dispersion Jposwve of positive 
example Images 

/ 

JposiHve = S ^ 5Z "^ii^ni " )^^i(^it " ) 
4=1 n=l 

14. A method as recited in claim 12, wherein computing a 
discrimination score includes computing the distance between said negative 
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example Image and said each database Image; said highest discrimination 
score corresponding to the lowest of said distance between said negative 
example Image and said each database Image. 

15. A method as recited In claim 14, wherein said at least one 
relevant feature includes a number / of relevant features; said positive example 
image Is the weighted average x} of Ni positive examples for each relevant 
feature /; being defined by: 

wherein ttI is a relevance degree for the positive example n; 

said negative example image Is the weighted average W of N2 negative 

examples for each relevant feature /; W being defined by: 

wherein yrS is a relevance degree for the negative example n; 
said distance between said positive example Image and said each database 
image represented by feature vector x„, minus said distance between said 
negative example image and said each database image represented by feature 
vector being defined by- 

wherein U/ is the global weight assigned to the i**' relevant feature ; and 

Wi is a symmetric matrix that allows to define the generalized 
ellipsoid distance D;and ui and W; minimizing the internal dispersion of positive 
example Images, minimizing the Internal dispersion of the negative example 
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images, and maximizing the discrimination between tlie positive and tlie 
negative examples. 

16. A method as recited in claim 15, wherein minimizing the 
Internal dispersion of positive example images, minimizing the internal 
dispersion of the negative example images, and maximizing the discrimination 
between the positive and the negative examples Is achieved by minimizing A/R 
where: 

where k = 1 for positive example and Ic = 2 for negative example, and where q, 
is the Weighted average of all positive and negative example images for the i* 
feature and is defined by 



E2 -^Nk -k^k 
_ fc=l 2^71=1 ^n^ni 

2L.k=\ 2^n-l 

17. A method as recited In claim 2, wherein said positive and 
negative example images are selected by a person among a list of sample 
images. 

18. A content-based method for retrieving data files among a 
set of database files, the method comprising: 

providing positive and negative example of data files; said 
positive example Image including at least one relevant feature; 
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restricting the set of database files to a subset of files selected 
among said database files; each files in said subset of files being selected 
according to its sioiildrlty with said positive example based on said at least one 
relevant feature; 

retrieving files in said subset of files according to their similarity 
with said positive example based on said at least one relevant feature and 
according to their dissimilarity with said negative example based on at least 
one discriminating feature between said positive and negative examples; 
whereby, the files retrieved among said database files corresponding to files 
similar to said positive example and dissimilar to said negative example. 

19. A content-based method for retrieving images among a set 
of database Images, the method comprising: 

providing positive and negative example images; said positive 
example image including at least one relevant feature; 

restricting the set of database images to a subset of Images 
selected among said database images; each images in said subset of Images 
being selected according to Its similarity with said positive example based on 
said at least one relevant feature; 

retrieving images in said subset of images according to their 
similarity with said positive example based on said at least one relevant feature 
and according to their dissimilarity with said negative example based on at 
least one discriminating feature between said positive and negative examples; 
whereby, the images retrieved among said database images corresponding to 
images similar to said positive example and dissimilar to said negative 
example. 

c 

20. A content-based system for retrieving images among a set 
of database images comprising: 
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means for providing positive and negative example images; said 
positive example image including at least one relevant feature; 

means for providing at least one discriminating feature in at least 
one of said positive and negative examples allowing to differentiate between 
said positive and negative example images; 

means for computing, for each database image in said set of 
database Images, a relevance score based on a similarity of said each 
database image to said positive example image considering said at least one 
relevant feature; 

means for creating a list of relevant images comprising the Nbi 
images having the highest similarity score among said set of database images; 
Nbi being a predetermined number; 

means for computing, for each relevant image in said list of 
relevant Images, a discrimination score based on a similarity of said each 
relevant image to said positive example Image considering said at least one 
discriminating feature and on a dissimilarity of said each relevant image to said 
negative example image considering said at least one discriminating feature; 
and 

means for selecting the Nb2 images having the highest 
discrimination score among said list of relevant images; Nba being a 
predetermined number. 

21. A system as recited in claim 20, wherein said means for 
providing positive and negative example images includes a graphical user 
interface displaying sample images. 

22, A system as recited In claim 20, wherein said graphical 
user interface includes means for specifying the degree of relevance of each 
said sample images. 
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23. A system as recited in claim 22, wherein said grapliical 
user interface includes means for viewing the retrieved images. 

24. An apparatus for retrieving images among a set of 
database images, the apparatus comprising: 

an interface adapted to receive positive and negative 
example images; said positive example image including at least one relevant 
feature; 

a restriction component operable to restrict the set of 
database images to a subset of images selected among said database images; 
said Images in said subset of images being selected according to their similarity 
with said positive example based on said at least one relevant feature; 

a retrieval component operable to retrieve images in said 
subset of images according to their similarity with said positive example based 
on said at least one relevant feature and according to their dissimilarity with 
said negative example based on at least one discriminating feature between 
said positive and negative examples; 

whereby, the images retrieved among said database images correspond- to 
images similar to said positive example and dissimilar to said negative 
example. 

25. An apparatus according to claim 24, wherein the restriction 
component and the retrieval component are implemented within the same logic 
device. 

26. A computer readable memory comprising content-based 
image retrieval logic for retrieving images among a set of database images, 
the content-based image retrieval logic comprising: 
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image reception logic operable to receive positive and 
negative example Images; said positive example image including at least one 
relevant feature; 

restriction logic operable to restrict the set of database 
images to a subset of images selected among said database Images; said 
images In said subset of images being selected according to their similarity with 
said positive example based on said at least one relevant feature; and 

retrieval logic operable to retrieve images in said subset of 
images according to their similarity with said positive example based on said at 
least one relevant feature and according to their dissimilarity with said negative 
example based on at least one discriminating feature between said positive and 
negative examples; 

whereby, the images retrieved among said database images conrespond to 
images similar to said positive example and dissimilar to said negative 
example. 
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